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System and Method for Camera Calibration 

Appendix on a CD: 

Attached to this application is a CD (i.e. a compact disk) that includes text files (that is, an 
5 octal dump) of the programs listed below plus several text files. 

a) A calibration program used to determine the parameters needed to seam images. 

b) A seaming program which will seam images using the parameters determined by the 
calibration program. 

c) Text files with a copyright notice, operating instructions and an example of camera 
10 calibration parameters. 

Field of the Invention: 

The present invention relates to cameras and more particularly to a system and method 
j 5 for determining the characteristics of cameras. 

m 15 

;^ Background of the Invention: 

ill Panoramic images can be created by seaming together multiple single view images. Multi 

!=| lens cameras are available which can simultaneously capture multiple single view images. 

The images captured by such a camera can be subsequently seamed into a panoramic 

;*. tksr/iv 

j r 20 image. Patent application serial number 09/602,290 filed 6/23/00 describes an interactive 
m seamer for creating panoramic images. Patent application serial number 09/338,790 filed 

! S 6/23/99 describes a multi-lens cameras. The disclosure in the referenced patent 

□ application is hereby incorporated herein by reference. 

25 It is practically impossible to obtain a set of single view images that fit together perfectly 
into a panorama. Parameters such as the heading, pitch, bank, offset, brightness and 
contrast of each image must be adjusted in order to make the images fit together into a 
panorama. One of the reasons that the parameters of the individual images must be 
adjusted is that the individual camera elements in the cameras that recorded the images 

30 have their own peculiarities and idiosyncrasies. In general each individual camera and 
each individual camera lens has parameters and idiosyncrasies that are peculiar and 
particular to that lens alone 
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If one determines the adjustments that will make to one set of images (recorded by a 
particular camera) fit together to form a panorama, these adjustments will in general 
provide an acceptable modification for a different set of images from the same camera so 
that the second set of images will fit together into a panorama. 

5 

Summary of the invention: 

The present invention provides a system and method for determining the adjustments that 
must be made to the individual images recorded by a multi-lens camera so that the 
images will fit together into a panorama. With the present invention the camera is placed 

1 0 in the center of a structure, the edges of which have at least one row on indicia. Each 
camera records an image which includes the indicia along each edge that will be seamed 
with an edge of another image in order to form a panorama. Each edge of the structure 
which is along a seam that will-be seamed with an edge of another image is recorded by 
two lenses on the camera. The amount of adjustment required to make the indicia in the 

1 5 two images fit together is then determined. The values of these adjustments are recorded 
along with an identification of the camera which recorded the images. Subsequent sets of 
images recorded by this same camera can then be seamed using the same adjustments. 

Another feature of the present invention includes providing two rows of indicia along the 
20 edges of the cube. The two rows of indicia are spaced apart by an amount equal to the 
distance between the lenses on the camera. This eliminates the need to correct for 
parallax and eliminates the need to position the camera at the exact center of the cube. 
Another feature of this invention is that the shape of the indicia is distorted so that when 
the indicia are recorded through a fisheye lens, the dots appear un-distored, thereby 
25 facilitating exact alignment of the images. 

Brief Description of the Drawings: 

Figure 1 is an overall process flow diagram. 
Figure 2A is an overall diagram of the calibration cube. 
30 Figure 2B is a more detailed view of a portion of the calibration cube. 

Figure 2C is an illustration showing why the rows of dots are spaced apart. 
Figure 3A shows an image recorded through a single lens on the camera. 
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Figure 3B shows two images displayed side by side, so that corresponding dots can be 
identified. 

Figure 3C shows a blank unit sphere and the default parameters used to initially seam 

images into a panorama. 
5 Figure 4 shows multiple images seamed together using default parameters and the rows 

of dots do not coincide. 
Figure 5 illustrates the distortion to dots caused by a lens. 
Figure 6 is a flow diagram showing the operation of the program. 
Figure 7 is a flow diagram of an alternate embodiment of the invention. 

10 

Detailed Description: 

An overall flow diagram of the process performed by a preferred embodiment of the 
invention is shown in Figure 1. The calibration structure used is shown in Figures 2A and 
j 2 2B. The calibration structure 200 is cube shaped. First, as indicated by block 101 a 

ill 1 5 particular camera 21 3 is used to capture images of the edges of the calibration cube 200. 
;H The images so captured can be termed calibration images. 

\t The camera is a multi-lens camera with one lens pointing toward each side of the cube 

200. A detailed description of camera 21 3 is given in co-pending application serial 
[!: 20 number 09/338,790 filed 06/23/99 entitled "System for Digitally Capturing and Recording 
ill Panoramic Movies" the specification of which is hereby incorporated herein by reference. 

Ill The camera 21 3 has six lenses and six images, one of each side of cube 200, are 

Q simultaneously recorded. Two lenses of camera 213 capture an image of each edge of 

the cube 200. 

25 

Figure 3A shows an example of an image 331 captured by one lens of camera 21 3. At 
the edge of image 331 (and at the edge of each image captured by camera 213) there are 
two rows of dots corresponding to the dots along one edge of cube 200. Since two lenses 
capture each edge of the cube 200, for each edge of cube 200, there are two images that 
30 show the same row of dots. 

After the images have been recorded, the operator manually indicates to the program (as 
indicated by block 102) which dots in the various images correspond to each other. 
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During this step, the two images which show the same edge of the cube are displayed 
adjacent to each other as shown in Figure 3B. The operator manually identified to the 
program which dots correspond in the two images. The operator does this by first clicking 
(with a mouse) on a "start" button and by then "clicking" on the two corresponding dots in 
5 sequence. The program recognizes the sequential "clicks" as indicating that the dots 
correspond. 

Next as indicated by block 104, the operator finds the settings of various parameters 
needed in order to align the images so that they can be seamed into a panorama without 
10 artifacts. The correction factors found are recorded along with an identification of the 
camera that created recorded the images. In the first preferred embodiment, the 
parameters of the camera that are determined for each of the six camera lenses are: 



d) amount of pin cushion distortion in the camera lens, 

e) amount of pitch, roll, heading and offset. 

In the following discussion, when reference is made to camera parameters, it is the above 
parameters that are being discussed. It is however, noted that the invention could be 
20 extended to other camera parameters (or some could be eliminated) as necessary for 
particular applications. 

A camera that went through the above process can then be used to record some other set 
of images and the parameters determined as described above can be used to seam this 
25 new set of images. That is, the final step 105 shown in Figure 1 is performed on a new 
set of images which were recorded with a previously calibrated camera. At this point the 
previously determined correction factors are applied to the new set of images. 

Figure 2A is an overall diagram of the calibration cube 200 showing the twelve edge 
30 panels designated 201 to 212. A camera 213 is positioned approximately at the center of 
the cube. All twelve panels 201 to 212 are identical and thus only panel 202 will be 
discussed in detail. Figure 2B shows panel 202 and the panels connected to it in more 
detail. Panel 202 has two rows of dots designated 202A and 202B positioned on the 



a) camera aiming which includes heading pitch and bank, 

b) field of view, 

c) offset of the camera lens to the CCD, 




panel. The panels are positioned at a 45 degree angle relative to the sides of the cube. If 
a line were drawn from the center of each panel (for example from point 202C) 
perpendicular to the surface of the panel the line would go to the center of camera 213. 

5 When activated, camera 213 records an image of each side of the cube 200. Each image 
so recorded includes the four panels surrounding the sides of the cube which is recorded. 
An example of such an image is shown in Figure 3A. It is important to note that each 
edge panel is recorded by two lenses from camera 213. Thus, each edge of the cube 
appears on two images. 

10 

The distance between the two rows of dots on each panel matches the distance between 
the lenses on camera 213. This is done in order to eliminate the need for any adjustment 
due to parallax. The reason for this is illustrated in Figure 2C which illustrates two lenses 

!*! 21 3A and 21 3B on camera 213 and the two dots on panel 202A and 202B on panel 202. 

ill 15 Since the dots are spaced the same distance apart as the lenses which recorded the dots, 
there is no need for a parallax adjustment to take into account the distance of the camera 

£i 213 from the panel 202. Naturally the program that matches images taken by the two 

In lenses must take into account that the two rows of dots are separated by a certain amount 

of distance. 

5 20 

Hi Figure 3B illustrates how the operator identifies to the program which dots in adjacent 

images correspond. The images illustrated in Figure 3B are an example of the images 

□ used in the operation indicated by block 102 in Figure 2. As illustrated in Figure 3B, the 

program displays two images 331 and 332 side by side so that the operator can see at 
25 least a portion of both rows of dos in both images. For example Figure 3B shows the 
image taken by lens 213A next to the image taken by lens 21 3B. The rows of dots on 
panel 202 appear in both images. The program includes shape recognition logic that 
locates and records in a table the position of each dot. 

30 An example of the main elements of such a table are given in Table One below. 
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image 1 D # 


Dot ID # 


X position 


Y position 


Cor-Dot ID 


IM0097 


D301 


456 


237 


D302 


IM0098 


D302 


159 


857 


D301 























10 



TABLE ONE 

In the above table, each row pertains to a particular dot. The columns in the table give the 
following information: 

Image ID # A unique number that identifies a particular image 

Dot ID # A unique number that identifies a particular dot 

x Position The x position of the particular dot 

y Position The y position of the particular dot 

Cor-Dot ID The Dot IDnumber of a corresponding dot on a different image. 

The information in this column is entered after the operator 
identified to the program which dots in the two images correspond. 



When as illustrated in Figure 3B, the images are displayed next to each other an operator 
can easily visually identify the corresponding dots. For example dot 301 corresponds to 
dot 302. After clicking on a "start" button to initiate the operation, the operator first "clicks" 
15 on dot 301 and then the operator clicks on dot 302. This tells the program that these two 
dots correspond and the program records in the last column of the above table the fact 
that these two dots correspond. The operator sequentially does this for each dot along 
edge 202P. The process is repeated for each of the twelve edges of the cube 200. 

20 It is noted that each images does not need to contain a complete image of both rows of 
dots. For example, the bottom of image 331 shown in Figure 3A and 3B does not contain 
a complete image of both rows of dots. The two adjacent dots (for example dots 271 and 
272 in Figure 2B) can be considered as constituting a hypothetical bar or area that begins 
at one dot and ends at the other dot. When the operator sequentially clicks on dots 301 

25 and 302 (see Figure 3B), the operator is identifying to the program the two "hypothetical 
bars" or area contain corresponding dots in the two images. Since each image shows at 
least the inside row of dots on each panel, and the distance between the rows of dots is 
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known to the program, the program can determine the location of all dots, even if they are 
not explicitly shown in the image. 

Next the program seams the images into a panorama using a set of default parameters. 
5 When the program is initiated it brings up the window shown in Figure 3C. The window 
shows a panorama with no images loaded. On the right side of the screen are various 
adjustments that can be made to images which are to be seamed into a panorama. 
These parameters are initially set to a set of default parameters such as shown below. 
The six cameras are identified as the Top, Front Bottom Back Left and Right cameras. 
10 The Field of View (FOV), Heading, Pitch, Bank, x Offset and y Offset for the cameras are 
initially set to the values shown in the following table. It should be understood that the 
following initial values are only examples and the values for various types of cameras can 
be determined by experimenting to find which give the best results. 



Camera 


FOV 


Heading 


Pitch 


Bank 


x offset 


y Offset 


Top 


95 


180 


90 


0 


0 


0 


Front 


95 


0 


0 


0 


0 


0 


Bottom 


95 


0 


-90 


0 


0 


0 


Back 


95 


180 


0 


0 


0 


0 


Left 


95 


-90 


0 


0 


0 


0 


Right 


95 


90 


0 


0 


0 


0 



Table Two 



Provision is also made for other parameters that can be used if desired. For example the 
20 amount of Fisheye adjustment on pin cushion adjustment can be set. 

The images as seamed using the default parameters are shown to the operator as shown 
in Figure 3C. As shown in Figure 4, the images have been seamed, but the dots on the 
various images do not correspond. For example, dot 304D from image 304 should 
25 coincide with dot 305D from image 305. However, the default parameters which produced 
the panorama shown in Figure 4 did not seam the images as they should be and hence 
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the dot 304D and dot 305D do not coincide. The operator can then change the various 
parameters used during the seaming operation in order to make the dots coincide. This is 
similar to the manual operation described in co-pending application serial number 
09/602,290 filed 6/23/2000 (the specification and drawings of which are hereby 
5 incorporated herein by reference). 

Figure 6 is a program flow diagram showing in detail how the program performs the 
operations indicated by blocks 102, 104 and 105 in Figure 1. That is, Figure 6 is a flow 
diagram showing how the corresponding dots on a set of images are identified, the 
10 appropriate correction factors are determined to make the dots coincide (i.e. the 

appropriate correction factors are determined in order to properly seam the images and 
finally how these parameters are used to seam another set of images recorded by the 
same camera. 

15 First as indicated by block 601, the six bitmaps (i.e. the six images) recorded by a 

particular camera 213 are loaded into the program. These images show the edges of 
calibration cube 200. Each edge appears in two of the images. Next as indicated by 
block 602 a conventional shape recognition routine is used to identify the location of the 
center of the dots at the periphery of each image. 

20 

The operator next identifies corresponding dots to the program as indicated by block 603. 
The operator can for example visually determine that dots 301 and 302 shown in Figure 
3B are in fact the same dot on the frame edge and that these dots should coincide. The 
operator uses a mouse to click in sequence on the corresponding dots. The program 
25 recognizes the clicks as indicating that the two dots on which the operator has clicked 
correspond. The identity of the corresponding dots are then recorded in a table such as 
the previously give Table One. Corresponding dots are termed alignment points. 

Next as indicated by block 605, the alignment points are mapped onto a sphere of unit 
30 radius. The mapping is done using a default set of parameters for camera direction, field 
of view, lens distortion, etc. The heading and pitch of each alignment point on the sphere 
is determined. Zero pitch and zero heading is taken as perpendicular to the center of 
each image. 
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The next series of steps 606A to 606D is at the heart of the calibration process. A least 
squares fitting process is used to change the various parameters so that the 
corresponding dots in the images coincide with each other. First as indicated by block 
5 606A a vector is established going between each pair of corresponding points. For 

example a vector going between 301 and 302 is Figure 3B is calculated. A similar vector 
is calculated for each pair of points. The length of the vector represents the amount of 
error in the alignment of the points. 

10 Next as indicated by block 606B, the amount of change in each vector which results from 
a small change in each lens parameter is determined. That is, each of the lens 
parameters such as heading, field of view etc. is changed by a small increment and the 
resulting change-in the length of the vectors is determined. The lens parameters are- 
changed one at a time and the resulting change in each vector is determined. 

15 

A conventional least squares fit for the error vectors is calculated to determine the 
changes in the camera parameters that reduce the error vectors to the smallest values. 
This is indicated by block 606C. Next the changes are applied to the parameters as 
indicated by block 606D. If the effect of changes in camera parameters on the error 

20 vectors was linear, it would only be necessary to go through steps 606A to 606D one 
time. However, the effect on the error vectors of changing the camera parameters is not 
linear, hence, steps 606A to 606D re executed three times as indicated by block 607. 
Each time steps 606A to 606D are executed the result converges closer to the optimum 
values. Naturally it will be understood by those skilled in the art that the selection of three 

25 times is arbitrary and based on experience and that a number is selected that produces a 
result of sufficient accuracy. 

After the process in blocks 606A to 606D has been repeated three times, the resulting 
panorama is displayed to the operator to insure that the process has been successful. If 
30 the panorama displayed is not satisfactory the operations are repeated. If the resulting 
panorama is acceptable, the operator saves the values of the parameters which 
generated the satisfactory panorama. 
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It has been found that if the program is given the freedom to change all the camera 
parameters, the result can be a relatively large change in the field of view from one 
particular camera to another particular camera. While this does get the image into 
alignment, it adds a certain amount of size distortion to the image. This potential problem 
5 can be avoided by giving the operator the ability to lockout (not use) some of the camera 
offsets during the least squares fit operation. Alternate embodiments provide a 
mechanism whereby the program automatically determines which offsets to lock during 
the least squares fit operation. 

1 0 Another feature of this invention is that the dots on the edges of the cube 200 are not in 
fact round. It is known that when a circle is viewed through a fisheye lens the image of 
the circle will not be round. The distortion produced by a fisheye lens can be understood 
by examining Figure 5. . Figure 5 shows a array of dots that has been distorted so that 
when viewed through a fisheye lens these dots will appear as a square array. The 

1 5 amount of distortion theoretically produced by a fisheye lens is know in the art. Each dot 
shown in Figure 5 has a shape such that when the dot is viewed through a fisheye lens, 
the dot appears round. This same type of distortion is provided for the dots on panels 201 
to 212. That is, each dot on the panels 201 to 212 has a shape so that when the dot is 
viewed through a fisheye lens, the dot appears as a round dot. . This facilitates the 

20 recognition and location of the dots by the program. It is noted that in 

Figure 5, the rows of dots are not straight, that is, the rows are concave. The shape of 
the individual dots on panels 201 to 212 is distorted as described above; however, the 
rows of dots are straight. Since the rows of dots on panels 201 to 212 are straight, the 
rows of dots in the images such as shown in Figure 3A are convex due to the action of the 

25 fisheye lens. Naturally, if the system used a lens type other than fisheye, the dots would 
have a distortion corresponding to the distortion produced by the particular type of lens 
used. 

In an alternate embodiment of the invention, a program is provided which automatically 
30 identifies the corresponding dots in adjacent images, this eliminates the need for the 
operator to identify corresponding dots to the program operator. 
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It is noted that the calibration structure used by the preferred embodiment is in the shape 
of a cube. The reason for this is that the camera 213 is in the shape of a cube. That is, 
the lenses on camera 213 are positioned on the sides of a cube. If the lenses where 
positioned differently, the structure would have a different shape. The term "edge which 
5 will be seamed" is used to mean an edge of an image which will be seamed with the edge 
of another image to create a panorama. The structure must have a shape so that (a) 
each edge which will be seamed is captured by two lenses, and (b) the image captured by 
each camera includes at least one row of dots along each edge which will be seamed. In 
the preferred embodiment, each edge panel 201 to 212 is captured by two lenses, and the 
10 image captured by each lens contains at least one row of dots along each edge. 

If the lenses of a camera are positioned in a configuration which differs from the positions 
0 f the lenses on camera 213, the calibration structure would have a different shape. The 
important point is that the calibration structure would have a shape such that (a) each 
j =( edge which will be seamed is captured by two lenses, and (b) the image captured by each 

ill 15 camera includes at least one row of dots along each edge which will be seamed. Many 
] L \ alternatives could be found which meet the above criteria. As a simple example, if the 

iji camera was a cube with two lenses on the top side of the cube, each of which captured 

;« 45 degrees, the calibration structure would be a cube with a panel at the middle of the top 

of the cube. 

:~ 20 

: — it 

ill It is noted that in the preferred embodiment the indicia along the edges of the cube are in 

!« the shape of dots which appear as round areas on the captured images (after they have 

□ been distorted by a fisheye lens). Alternatively, indicia with other shapes could be used. 

For example, the indicia on the panels 201 to 212 could be in the shape of bars. These 
25 bars could be pre-distorted so that the appear rectangular after being distorted by a 
fisheye lens. The pre-distortion is only needed to make the recognition logic in the 
program simpler; however, it is not required for the operation of the invention. A wide 
variety of indicia shapes could be used as alternatives to the shape used in the preferred 
embodiment. 

30 

The identification of the camera that recorded a particular set of images can be manually 
recorded and then this identification can be applied to a set of camera parameters 
determined as described above. In an alternate embodiment described with reference to 
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Figure 7, the multi lens camera automatically records on each image an identification as 
described in co-pending application serial number 60/218,604 filed 7/17/2000 (the 
specification and drawings of which is hereby incorporated herein by reference). It is 
noted that the images recorded by the cameras described herein are digital in nature. 
5 Hence the images recorded at one time can be stored in a file. The camera identification 
is placed in a header in this file. 

In this embodiment, the calibration program reads this camera identification from the file 
header of the file that contains the images. The calibration program then records the 

1 0 camera identification along with the calibration parameters for that particular camera. The 
result is a table showing camera identification numbers and corresponding calibration 
parameters. When a set of images is read by the seaming program, it automatically reads 
the camera identification from the file header of the file that contains the images. The * 
process is shown in Figure 7. First, as indicated by block 761 , a camera records an 

15 image of the calibration cube as described above. The single view images generated are 
recorded along with the camera identification as described in the above referenced patent 
application. Next as indicated by block 762. The camera calibration parameters are 
determined as described above. 

20 As indicated by block 765, a table is produced showing camera calibration parameters 
and camera identification. At a later time another set of images is recorded with this 
same camera as indicated by block 766. The camera identification is recorded along with 
the images. Finally, as indicated by block 767, when the images are sent to a seaming 
program, the camera identification is reads the camera identification along with the image 

25 data. The seaming program interrogates the table to find the correct parameters to use to 
seams the images using the appropriate parameters. 

This application includes an appendix submitted on a CD (i.e. a compact disk). The CD 
includes the text files (i.e. octal dumps) of executable programs and other text files as 
30 listed below: 

a) A calibration program used to determine the parameters needed to seam images. 

b) A seaming program which will seam images using the parameters determined by the 
calibration program. 
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c) Text files with a copyright notice, operating instructions and an example of camera 
calibration parameters. 

While the invention has been shown and described with respect to preferred embodiments 
5 thereof, it will be understood by those skilled in the art that various changes in form and 
detail can be made without departing from the spirit and scope of the invention. Many 
different embodiments of the applicant's invention are possible and the invention is not 
limited to only the embodiment described herein. Applicant's invention is limed only by 
the scope of the appended claims and equivalents thereof. 
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